Papers with human correlation
Difficulty-Aware Machine Translation Evaluation (2021.acl-short)
Copied to clipboard
| Challenge: | Current MT evaluation measures pay the same attention to each sentence component . in real-world examinations, the questions vary in difficulty and weightings . |
| Approach: | They propose a difficulty-aware MT evaluation metric that takes translation difficulty into account . they propose to use this metric to evaluate machine translation (MT) results . |
| Outcome: | The proposed method outperforms most MT evaluation metrics in terms of human correlation. |
Improving Image Captioning Evaluation by Considering Inter References Variance (2020.acl-main)
Copied to clipboard
| Challenge: | Existing one-to-one metrics penalize mismatches without considering the intrinsic variance between ground truth captions. |
| Approach: | They propose a one-to-one metric based on BERTScore that could be extended to include new features for image captioning evaluation. |
| Outcome: | The proposed metric achieves state-of-the-art human judgment correlation while improving performance. |
RoViST: Learning Robust Metrics for Visual Storytelling (2022.findings-naacl)
Copied to clipboard
| Challenge: | Visual storytelling is the task of generating a story paragraph that describes a given image sequence. |
| Approach: | They propose 3 evaluation metrics sets that analyze which aspects we would look for in a good story . they compare their correlation with human judgement scores on a sample of machine stories . |
| Outcome: | The proposed evaluation metrics outperform other metrics on human correlation on a sample of machine stories from state-of-the-art models. |
From Shijing to English and German: Resources and Evaluation for LLM Translation of Early Chinese Poetry (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) show promise in literary translation, but their performance in poetry remains unexplored. |
| Approach: | They propose a framework that integrates knowledge-driven, rule-based, and LLM-as-judge metrics into a Shijing corpus . their code, lexical KB, and corpus reconstruction protocols are available at https://github.com/ML-KULeuven/ShijingLLMTrans. |
| Outcome: | The proposed framework achieves higher human correlation than traditional metrics and high statistical stability. |
PrefIx: Understand and Adapt to User Preference in Human-Agent Interaction (2026.findings-acl)
Copied to clipboard
| Challenge: | Current benchmarks evaluate task accuracy but overlook how agents interact . Preference-aware agents show 7.6% average UX improvement and 18.5% gain in preference alignment. |
| Approach: | They propose a configurable environment that evaluates both what agents accomplish and how they interact. |
| Outcome: | The proposed model improves performance and improves user experience by 7.6% and 18.5% respectively. |